xarrayvideo

Save multichannel data from xarray datasets as videos to save up massive amounts of space (e.g. 20-50x compression) with minimal quality loss.

This library provides two functions: xarray2video to encode some xarray variables into videos, and video2xarray to rebuild the xarray from the videos. It can encode lossily or losslessly, supporting all video formats supported by ffmpeg (e.g., x265, vp9) at different bit depths (e.g., 8,10,12,16) as well as all GDAL’s supported image formats (e.g. JPEG2000) for frame-by-frame encoding (provided mostly as a comparison baseline).

Features

LIMITATIONS: As of now, everything is loaded in memory, which will be a problem for larger datasets. Future versions of this library will allow for indexing of frame ranges within the video file, as well as lazy loading.

Paper

If you find this library useful, please consider citing the accompanying paper:

TODO

Results

DeepExtremeCubes

An example xarray from the DeepExtremeCubes database is provided as an example. It consists of 495 timesteps of 128x128 Sentinel2 data sampled every 5 days, along with other segmentation maps.

Here is a plot with some results of different compression approaches for the multiespectral Sentinel 2 data (7 bands). Note that we also use JPEG2000 as a comparison, encoding every timestep at a time (but all bands 7 in a single image, instead of relying on sets of 3 bands as for video compression):

For this data, some conclusions can be reached. In summary: x265 > vp9 >> JPEG2000 and 16 > 12 > 10 >> 8 bits.

Example of compression (1.47% of original size). The quality loss is visually imperceptible.

Original (download for full size):

Compressed (download for full size):

These visualizations were generated using txyvis

Cesar’s cube

This test dataset consists of a single cube of 512x512 Sentinel 2 images of 102 timesteps, where every timestep corresponds to a month of observations, that was processed to exclude clouds. It contains 10 bands: 'B2', 'B3', 'B4', 'B5', 'B6', 'B7', 'B8', 'B8A', 'B11', 'B12'.

The conclusions are similar to the previous datacube:

About using PCA

For this data, we additionally compare the base approach of encoding all 11 channels in sets of 3 videos, with two other approaches:

The results for this tests are the following:

As can be seen, the base approach of encoding without PCA seems to peform best. This is likely due to PCA shifting the distribution of the data so that standard video codecs, not designed for this data, do not work opimally.

ERA5

We will 284 timesteps (period of 6h) of ERA5’s wind_u variable at its native resolution (1440x721x13), hence this being 13-channel dataset. For this test, we don’t include 8 or 10 bits results for faster results, since we saw that hihger bits are generally better.

In this case, the conclusions are different from the satellite data. Overall, JPEG200 seems to do better for all metrics considered, except for decoding time (which could be improved with parallel reading). It might be the case that video codecs fail to efficiently encode ERA5 data, which deviates significantly from the natural images that they were designed to encode.

Installation

#Install base requirements
pip install xarray numpy ffmpeg scikit-image scikit-learn pyyaml zarr netcdf4 ffmpeg-python gdal gcsfs openjpeg tqdm seaborn
#mamba install xarray numpy ffmpeg scikit-image scikit-learn pyyaml zarr netcdf4 ffmpeg-python gcsfs tqdm seaborn
#mamba install -c gdal-master gdal openjpeg

#[Optional] Requiremetns for temporal alignment of video slices
pip install satalign

#[Optional] Requierements for plotting (optional, but `plot_image` calls will fail)
pip install ipython opencv-python
pip install git+https://github.com/OscarPellicer/txyvis.git

#[Optional] Requierements for much quicker metrics
pip install torchmetrics
#mamba install torchmetrics

#Download repo
cd ~
git clone https://github.com/OscarPellicer/xarrayvideo.git
cd xarrayvideo

#Unzip the example xarray
!unzip cube.zip

Usage

To see some examples run jupyter lab or VSCode to open example.ipynb

The basic syntax (for the DeepExtremesCubes database) is the following:

#Load libraries
import xarray as xr
import numpy as np
from xarrayvideo import xarray2video, video2xarray, plot_image

#Define paths
array_id= '-111.49_38.60'
input_path= '../mc_-111.49_38.60_1.2.2_20230702_0.zarr'
output_path= './out'

#Load cube
minicube= xr.open_dataset(input_path, engine='zarr')
minicube['SCL']= minicube['SCL'].astype(np.uint8) #Fixes problem with the dataset
minicube['cloudmask_en']= minicube['cloudmask_en'].astype(np.uint8)

#Set up the compression params
lossless_params= { 'c:v':'ffv1' }
lossy_params= { 'c:v': 'libx265', 'preset': 'medium', 'crf': 1, 
                'x265-params': 'qpmin=0:qpmax=0.1:psy-rd=0:psy-rdoq=0  }
conversion_rules= {
    's2': ( ('B07','B06','B05','B04','B03','B02','B8A'), 
                ('time','x','y'), 0, image_lossy_params, 16),
    'scl': ( 'SCL', ('time','x','y'), 0, lossless_params, 8),
    'cm': ( 'cloudmask_en', ('time','x','y'), 0, lossless_params, 8),
    }
    
#Compress, with compute_stats it takes a bit longer, but shows compression info
arr_dict= xarray2video(minicube, array_id, conversion_rules,
                       output_path=output_path, compute_stats=True,
                       loglevel='verbose', save_dataset= True
                       )  
    
#Decompress
minicube_new= video2xarray(output_path, array_id)

#Plot RGB bands
plot_image(minicube, ['B04','B03','B02'], save_name='./out/RGB original.jpg')
plot_image(minicube_new, ['B04','B03','B02'], save_name='./out/RGB compressed.jpg')

Contact

Contact me: oscar.pellicer [at] uv.es or open an Issue